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Abstract —Cloud Computing Datacenters host millions of vir¬ 
tual machines (VMs) on real world scenarios. In this context, 
Virtual Machine Placement (VMP) is one of the most challenging 
problems in cloud infrastructure management, considering also 
the large number of possible optimization criteria and different 
formulations that could be studied. VMP literature include rele¬ 
vant topics such as energy-efficiency. Service Level Agreements 
(SLA), cloud service markets. Quality of Service (QoS) and 
carbon dioxide emissions, all of them with high economical and 
ecological impact. This work presents an extensive up-to-date 
review of the most relevant VMP literature in order to identify 
research opportunities. 

L Background and Motivation 

The process of selecting which virtual machines (VMs) 
should be located (i.e. executed) at each physical machine 
(PM) of a datacenter is known as Virtual Machine Placement 
(VMP). 

The VMP problem has been extensively studied in cloud 
computing literature and several surveys have already been 
presented. Existing surveys focus on specific issues such as: 
(1) energy-efficient techniques applied to the problem 0, 
GQl, (2) particular architectures where the VMP problem is 
applied, specifically federated clouds ll27l and (3) methods for 
comparing performance of placement algorithms in large on- 
demand clouds 16^ . 

Beloglazov et al. presented in 0 a survey of energy-aware 
resource allocation policies and scheduling algorithms consi¬ 
dering QoS. The following open challenges were identified 
considering energy-aware management of cloud computing 
datacenters: (1) development of fast energy-efficient algo¬ 
rithms for the VMP, considering multiple resources for large- 
scale systems with the ability to predict workload peaks to 
prevent performance degradation, (2) energy-aware optimiza¬ 
tion of virtual network topologies between VMs for optimal 
placement in order to reduce network traffic and thus energy 
consumed by the network infrastructure, (3) development of 
new thermal management algorithms to appropriately control 
temperature and energy consumption, (4) development of 
workload-aware resource allocation algorithms, considering 
that current approaches assume a uniform workload, and 
(5) decentralization and distributed approaches to provide 
scalability and fault-tolerance to the VMP problem resolution. 

Salimian et al. presented a review of different selection 
and placement algorithms for energy-efficient management of 
cloud computing datacenters GQ). Approaches for virtual and 


physical resources modeling, applied techniques and future 
work were identified for each studied article. Most relevant 
future work include: (1) investigation of VMP for multi-core 
architectures considering multiple resources, (2) consideration 
of dynamic thresholds for QoS and (3) development of in¬ 
telligent schemes according to workload and considering live 
migration. 

Gahlawat et al. proposed in lIZTlI a brief survey of the 
main cloud federation architectures and approaches considered 
for VMP problem formulation in this particular scenario. 
It is important to remember that cloud federation is the 
practice of voluntarily interconnecting cloud infrastructures of 
different Cloud Service Providers (CSPs), mostly to respond 
to workload peaks. 

Einally, Mills et al. presented in ll^ an objective method 
to compare performance of placement algorithms in large on- 
demand clouds, where 18 different algorithms for VMP were 
compared considering 39 variables such as: reallocation rate, 
user request rate, allocation rate and disk space utilization, just 
for cite a few. 

The above mentioned surveys and research articles focused 
into specific issues related to the VMP problem. To the best 
of the author’s knowledge, there is not existing research work 
that presents a general and extensive study of a large part of 
the VMP literature. This work presents an extensive up-to- 
date review of the most relevant VMP literature in order to 
identify research opportunities on this important and promising 
research area. 

II. Reviewed Literature 

A selection process of current research work was defined 
and performed in order to study a large part of the most 
relevant VMP literature for this survey. This section presents 
a detailed description of the literature selection process which 
is summarized in Eigure [T] 

A. Keywords Search 

The selection process of relevant articles started with a 
search of research articles from Google Scholar database 
[scholar.google.com] with at least one of the following selected 
keywords in the article title: (1) virtual machine placement, 
(2) vm placement, (3) virtual machine consolidation, (4) vm 
consolidation or (5) server consolidation. 


This keywords search step results in 446 research articles. 
A detailed list of the 446 resulting articles could be found in 
El, specifically in the worksheet named keywords search. 

B. Publisher Filtering 

Considering the large number of results from keywords 
search step, the literature selection process focused in research 
articles published in the following relevant publishers: (1) 
ACM, (2) IEEE, (3) Elsevier and (4) Springer. Percentage of 
articles per publisher in the studied universe is summarized in 
Eigure 

This publisher filtering step results in a reduction from 446 
to 172 (38.6%) research articles of the literature. A detailed list 
of the 172 resulting articles could be found in El . specifically 
in the worksheet named publisher filtering. 

C. Abstract Reading 

Considering the 172 resulting articles from the publisher 
filtering step, an abstract reading was performed in order to 
identify only the most relevant articles that specifically study 
the VMP problem. After the abstract reading, 93 research 
articles were selected from the VMP literature. A detailed list 
of the 93 resulting articles could be found in El . specifically 
in the worksheet named abstract reading. 

Einally, short papers (i.e. research articles with less than 
6 pages) were removed from the selected literature, resulting 
in 84 selected articles of the VMP literature for the detailed 
study presented in this survey. 

III. Literature Overview 

Consolidating multiple underutilized servers into a fewer 
number of non-dedicated servers can respond to the common 
problem of server sprawl in datacenters 1^ . The conso¬ 
lidation can be modeled as a variant of the bin packing 
problem (items to be packed are the VMs and bins are 
the PMs) in order to help resource management considering 
that until 2008, server consolidation exercise was primarily 
a manual process. A manual process of this nature is time 
consuming and depends on the subjective assessment of the 
decision maker. Particularly, in ll34l it is studied the server 
consolidation problem considering technically deprecated PMs 
into technically superior PMs, but this problem could also be 
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Figure 2. Percentage of articles per publisher in the studied 84 papers. 

studied as a VMP problem. Experimental tests are performed 
considering a maximum number of 1000 PMs for mapping into 
620 PMs, solving extremely large instances of the problem in 
a reasonable amount of time, i.e. 0.188 seconds against 6262.1 
seconds (1.74 hours) for the optimal solution. The proposed 
algorithm obtained solutions with a 1.5 ratio more PMs than 
the optimal solution for largest problem instances, i.e. 620 
PMs obtained against 411 PMs from optimal solution. 

If a VM is considered to be a three dimensional object 
(CPU, memory and storage) then the problem of placing the 
VMs over the PMs looks similar to the three dimensional bin 
packing problem, but they are not exactly the same. In 3D bin 
packing problem, a set of three dimensional objects (generally 
cuboids) are required to be placed inside three dimensional 
containers (also cuboids). The aim is to pack as many objects 
in the containers as possible, so that the number of containers 
required is minimized. While packing objects into a given 
container, two objects can be placed side by side or one on 
top of the other, but if we consider VMs as the objects then 
placing VMs side by side or one on top of the other is not 
a valid operation. This is because once a resource is utilized 
or occupied by a VM, it can not be reused by any other VM 
1^ . The VMP problem is actually similar to Vector Packing 
Problem which is also a NP-Hard problem. 

In ll8^ . VMs and PMs are represented as a d-dimensional 
vector where each dimension corresponds to one type of 
resource (e.g. CPU, memory and storage) and for each PM 
should be balanced along different dimensions. Operational 
power and thermal dissipation are also studied. Eor each ob¬ 
jective an efficiency metric was defined. Eor the resolution of 
the VMP formulation, authors proposed a Genetic Algorithm 
(GA). Genetic operators are: (1) selection and crossover, where 
a random process do not perform efficiently, so the selection 
and crossover is based on each of the three efficiency metrics 
resulting on better solutions, (2) mutation, where groups of 
VMs are randomly selected and eliminated. 

In uni are studied two payment plans in cloud computing 
markets: (1) reservation and (2) on-demand plans (Amazon 
EC2 and GoGrid). Considering that prices of resources in 
reservation plans are generally cheaper than in on-demand 
plans, in advance reservation should be applied as maximum 
as possible. On-demand plans only should be used when 


Figure 1. VMP literature selection process 










the reservation does not fully meet predicted workload. The 
proposed algorithm makes a decision to host a certain number 
of VMs on appropriate cloud providers obtaining an optimal 
solution from a stochastic integer programming (SIP) formu¬ 
lation considering two stages of decision making: (1) first 
stage defines the number of VMs provisioned in reservation 
phase, while (2) the second stage defines the number of VMs 
allocated in both utilization and on-demand phases. 

In Q, it is defined a new high-availability property for 
a VM; when a VM is marked as /c-resilient, it should be 
guaranteed that it can be relocated to a non-failed host without 
relocating other VMs as long as there are up to k PMs failures. 
Anti-location, anti-co-location and resources constraints are 
also proposed for the VMP formulation, considering the opti¬ 
mization of the problem in its dual space (dual optimization 
problem). The transformed formulation enable a CP engine to 
compute a solution in reasonable time on practical cases. 

When a client request comes, the infrastructure service 
provider needs to identify a PM to place the client VM so 
that the following objectives can be achieved: (1) the SLA 
associated with the VM is satisfied, and (2) the performance 
metrics of SLAs associated with the VMs previously running 
on the target physical node is least affected ca. The proposed 
prediction contains the following initiation steps: (1) collect 
the performance data of a given application during its previous 
executions and treat it as foreground performance metrics, 

(2) collect the performance data of candidate physical nodes 
and treat the data as background performance metrics, and 

(3) perform Canonical Correlation Analysis (CCA) on the 
performance data collected in the previous two steps and 
produce the canonical weight vectors and scores for the most 
correlated patterns. 

A formal model for the application (to better understand 
structural constraints) and the (hierarchical) datacenter (to 
effectively capture capabilities) is proposed in ll42ll . Demand, 
communication and availability constraints are considered for 
the VMP formulation, considering experiments for a small 
datacenter (60 PMs) and a large one (200 PMs). 

Services offered by cloud providers such as Amazon, 
Microsoft, IBM, and Google are implemented on thousands 
of servers spread across multiple geographically distributed 
datacenters. There are at least three reasons behind this 
geographical distribution: the need for high availability and 
disaster tolerance, the sheer size of the computational infras¬ 
tructure, and the desire to provide uniform access times to the 
infrastructure from widely distributed user sites Eol. 

In ED, it is implemented a two-phase optimization process. 
During the first phase a subset of PMs with properties that best 
serve the VM placement is selected. The goal in selecting 
a subset of all available PMs is to reduce the number of 
constraints and the search space during the second phase. In 
Ga, it is proposed a framework for: (1) being able to add or 
modify a constraint without changing the algorithms and (2) 
being able to test and activate a new algorithm without having 
to re-implement every constraint within it. 


IV. Optimization Approaches 

This section presents the optimization alternatives proposed 
in the studied articles according to the optimization approach 
of the studied objective functions. The identified optimiza¬ 
tion approaches may be classified as: (1) mono-objective 
optimization problem (MOP), (2) multi-objective solved as 
mono-objective (MAM) and (3) pure multi-objective (PMO). 
The mentioned optimization approaches are detailed in the 
following sub-sections and summarized in Table [I] 

A. Mono-Objective Approach 

A mono-objective approach considers the optimization of 
only one objective or the individual optimization of more than 
one objective function, one at a time. 

According to the studied articles, the research of the VMP 
problem has been mainly guided by the mono-objective opti¬ 
mization approach considering that 61.9% of the studied arti¬ 
cles proposed a mono-objective approach (MOP) for solving 
the VMP problem. From the 52 articles that studied the VMP 
problem in a mono-objective approach, almost 40 different 
objective functions were proposed. It is also remarkable that 
some works studied the same objective functions but pro¬ 
posed different modeling approaches (e.g. economical revenue 
maximization could be achieved by minimizing the total 
economical penalties for SLA violations ifThll . by minimizing 
operational costs lf39l . Il40ll or even by maximizing the total 
profit for leasing resources ll73ll L 

Considering the large number of proposed objective func¬ 
tions and different approaches for modeling an objective 
function, multi-objective optimization m could result in 
better and more realistic formulations of the VMP problem, 
optimizing more than just one objective function at a time (e.g. 
achieve economical revenue maximization by simultaneously 
minimize the total economical penalties for SLA violations, 
minimize operational costs and maximize the profit for leasing 
resources). 

B. Multi-Objective solved as Mono-Objective Approach 

In this work, the optimization of multiple objective func¬ 
tions combined into one objective function is considered a 
multi-objective approach solved as mono-objective (MAM). 
A disadvantage of this approach is that requires a deep know¬ 
ledge of the problem domain to allow a correct combination 
of the objective functions, which in most cases is not possible 

m. 

According to this survey, 34.5% of the studied articles pro¬ 
posed a multi-objective approach but finally solved the VMP 
as a mono-objective optimization problem. In the last years, 
a growing number of articles have proposed formulations of 
the VMP problem with this hybrid approach (see Figure [^. 

The most popular method for solving a MAM problem is 
the Weighted Sum Method El, Q, HI, El, 0, EQ), EH, 

m, Gl, m, ED, M, US, EH, El, IMl, Ei, El, 

El- In this method one objective function is formulated as a 
linear combination of multiple objectives. 


Solving a multi-objective optimization problem as mono¬ 
objective considering the other objective functions as cons¬ 
traints of the problem is also a studied method in the VMP 
literature CIl, ESI. 

Kord et al. proposed a two-objective approach and made 
a trade-off between these two goals using a fuzzy Analytic 
Hierarchy Process (AHP) ||49]| . Other research articles em¬ 
ployed fuzzy logic to provide an efficient way for combining 
conflicting objectives and expert knowledge. For multiple 
objectives, fuzzy logic allows mapping of different objectives 
into linguistic values characterizing levels of satisfaction Ga, 

El, IM). 

C. Pure Multi-Objective Approach 

A general pure multi-objective optimization problem 
(PMO) includes a set of p decision variables, q objective 
functions, and r constraints. Objective functions and 
constraints are functions of decision variables. In a PMO 
formulation, x represents the decision vector, while y 
represents the objective vector. The decision space is denoted 
by X and the objective space as Y. These can be expressed 
as 114]: 

Optimize: 

y = f{x) = [fi{x),f 2 {x), ..., fg{x)] (1) 

subject to: 

e{x) = [ei{x),e 2 ix), ..., er{x)] > 0 (2) 

where: 

x =[xi,X 2 j..;Xp]€ X (3) 

y = [yi,y 2 ,-;yq]&Y (4) 

It is important to remark that optimizing, in a particular 
problem context, can mean maximizing or minimizing. The 
set of constrains e{x) > 0 defines the set of feasible solutions 
Xf C X and its corresponding set of feasible objective vectors 
Yf C Y. The feasible decision space Xf is the set of all 
decision vectors x in the decision space X that satisfies the 
constraints e{x), and it is defined as: 

Xf = {x\x € X Ae{x) >0} (5) 

The feasible objective space Yf is the set of the objective 
vectors y that represents the image of Xf onto Y and it is 
denoted by: 

Yf = {y\y = f{x) Vx G Xf} (6) 

To compare two solutions in a multi-objective context, 
the concept of Pareto dominance is used. Given two feasible 
solutions u, V G X, u dominates v, denoted as u A v, if 
f{u) is better or equal to f{v) in every objective function 
and strictly better in at least one objective function. If neither 
u dominates v, nor v dominates it, u and v are said to be 
non-comparable (denoted as it ~ u). 


Table I 

Optimization Approaches 


Optimization Approach 

Percentage 

References 

Mono-Objective (MOP) 

61.9% 

Ga, ED- (m ca isi 

(29), (59), (2), (83), (HI, 

CD, ED, (68), (ID, (38), 

(ill, ED, 03, 03, ED, 

(65), ED, (80), da, (39), 

ED, EH), ED, (43), ESI, 

(3, iSl 03, (30), (69), 

ED (23, ED ED, El 

0, (63, na, E3, (23, 

ED, (66), (60), (23. (13, 

ED, E3 

Multi-Objective solved 
as Mono-Objective 
(MAM) 

34.5% 

E3, mi, (D, (13, (D, Ei, 
ED, E3, (3, ED, ED 
E3, (23, E3, (23, ED 

(13, (36), (B), E3, ED, 

ED, Ea, ED, 113 (3 
E3, ED, (13 

Pure Multi-Objective 
(PMO) 

3.6% 

E3, ED, El 


25 



Year 

Figure 3. Articles per year according to each considered optimization 
approach from the universe of 84 papers. 

A decision vector x is non-dominated with respect to a set 
U, if there is no member of U that dominates x. The set of 
non-dominated solutions of the whole set of feasible solutions 
Xf, is known as optimal Pareto set P*. The corresponding set 
of objective vectors constitutes the optimal Pareto front PF*. 

In the considered literature, only 3.6% of the articles 
proposed a multi-objective approach, using Pareto Dominance 
for comparing conflicting objective functions 1581 . l5^ . 1^ . 
To the best of our knowledge, there is no many-objective 
optimization proposed for the VMP problem in literature (i.e 
a multi-objective optimization problem with more than three 
objective functions El)- 

V. Objective Functions 

In cloud computing datacenters with a considerable amount 
of PMs and VMs, there are many criteria that can be 
















Table II 

Objective Functions - Energy Consumption Minimization Group 


Objective Function 

References 

Datacenter Power Consumption Minimiza¬ 
tion 

(H 

Energy Consumption Minimization 

(n, (m, (u, (92), 

(231. (20), (ID, (83), 

(13, (23, (13, (B), 
(ID, 03, 03 OS), 
(ID, (2), ED, E3 

Energy Efficiency Maximization 

(23 

IP Layer Power Consumption Minimization 

E3 

Network Power Consumption Minimization 

03, OSl, 03, 03, 
ES 

Number of PMs Minimization 

03, (3 03, 03, 
(23, (2D, 03, 03, 
E3, 03 03, 03 

Power Consumption Minimization 

03, 03, (B, 03, 
03, OSl 03 03, 
03, 03, 03, (89) 

WDM layer Power Consumption 

(23 


considered when selecting a possible solution for the VMP 
problem, depending on policies and optimization objectives. 
These criteria can even change from one period of time to 
another, which implies a variety of possible formulations of 
the problem and different objective functions to be optimized. 

The studied articles proposed 56 different objective func¬ 
tions (Tables [I^ - |VI[ ) for the three optimization approaches 
presented in Section |IV] Considering the large number of 
proposed objective functions, this work classifies objective 
functions with similar goals into 5 objective function groups 
that are presented in following sub-sections. 

A. Energy Consumption Minimization 

Considering the studied articles, 50.0% proposed energy 
consumption minimization for the VMP problem considering 
different energy modeling approaches which are detailed be¬ 
low. Objective functions classified in this optimization group 
are; (1) Datacenter Power Consumption Minimization, (2) En¬ 
ergy Consumption Minimization, (3) Energy Efficiency Maxi¬ 
mization, (4) IP Layer Power Consumption Minimization, (5) 
Network Power Consumption Minimization, (6) Number of 
PMs Minimization, (7) Power Consumption Minimization and 
(8) WDM layer Power Consumption (see Table 0- 

B. Network Traffic Minimization 

Prom the studied articles, 30.9% proposed network traffic 
minimization for the VMP problem. As presented in section 
network traffic optimization could be studied jointly with 
energy consumption. Objective functions classified in this opti¬ 
mization group are: (1) Average Traffic Latency Minimization, 

(2) Cloud QoE Maximization (Response Time Minimization), 

(3) Cloud Service Response Time Minimization, (4) Data 
Access Minimization, (5) Data Transfer Time Minimization, 
(6) End-to-end Delay Minimization, (7) Link Congestion Min¬ 
imization, (8) Migration Number Minimization, (9) Migration 


Table III 

Objective Functions - Network Traffic Minimization Group 


Objective Function 

References 

Average Traffic Latency Minimization 

(23 

Cloud QoE Maximization (Response Time 
Minimization) 

ED 

Cloud Service Response Time Minimization 

03 

Data Access Minimization 

(3 

Data Transfer Time Minimization 

(SD, (23 

End-to-end Delay Minimization 

(13 

Link Congestion Minimization 

ED 

Migration Number Minimization 

(3, E3 

Migration Overhead Minimization 

(3 

Migration Time Minimization 

E3 

Network Cost Minimization 

(13, (13, (23 

Network Performance Maximization 

(23 

Network Traffic Minimization 

ED, (13, (SD (n), 
ED, E3 03 

Network Utilization Minimization 

ES) 

Node Cost Minimization 

E3 

Overall Communication Cost Minimization 

(23 

WAN Communication Minimization 

03 

Worst Case Cut Load Ratio Minimization 

E) 


Overhead Minimization, (10) Migration Time Minimization, 
(11) Network Cost Minimization, (12) Network Performance 
Maximization, (13) Network Traffic Minimization, (14) Net¬ 
work Utilization Minimization, (15) Node Cost Minimization, 
(16) Overall Communication Cost Minimization, (17) WAN 
Communication Minimization and (18) Worst Case Cut Load 
Ratio Minimization (see Table Uni- 

Modeling and quantification of live migration have been 
studied considering the minimization of migration number as 
well as the minimization of the overhead and migration time 
(see Table m- Live migration it self can be considered a very 
challenging future direction in the VMP context, considering 
it as a enabling technology for cloud computing management 
systems. 

C. Economical Costs Optimization 

Prom the studied articles, 22.6% proposed economical 
costs optimization for the VMP problem. Objective functions 
classified in this optimization group are: (1) Economical 
Revenue Maximization, (2) Electricity Cost Minimization, (3) 
Operational Cost Minimization, (4) Reservation Cost Mini¬ 
mization, (5) Server Cost Minimization, (6) SLA Violations 
Minimization, (7) Thermal Dissipation Costs Minimization 
and (8) Total Infrastructure Cost Minimization (see Table HYI- 

D. Performance Maximization 

In this work, objective functions that proposed formulations 
related to performance represent 16.7% of the studied articles. 
Objective functions classified in this optimization group are: 
(1) Availability Maximization, (2) CPU Demand Satisfaction 
Maximization, (3) Deployment Plan Time Minimization, (4) 


































Table IV 

Objective Functions - Economical Costs Optimization Group 


Objective Function 

References 

Economical Revenue Maximization 

Cl, ED, (H, CD, 

(36), (73) 

Electricity Cost Minimization 

(50) 

Operational Cost Minimization 

(39), (40) 

Reservation Cost Minimization 

CD 

Server Cost Minimization 

CD 

SLA Violations Minimization 

(92), CD, Cl, (49), 
(67), Cl 

Thermal Dissipation Cost Minimization 

(H 

Total Infrastructure Cost Minimization 

(6D 


Table V 

Objective Functions - Performance Maximization Group 


Objective Function 

References 

Availability Maximization 

(59), (H, (H, d 

CPU Demand Satisfaction Maximization 

(D 

Deployment Plan Time Minimization 

ED 

Peiformance Maximization 

(13, (13, (13, (H 

QoS Maximization 

ED 

Resource Interference Minimization 

ED 

Security Metrics Maximization 

(H 

Shared Last Level Cache (SLLC) Con¬ 
tention Minimization 

(H 

Total Job Completion Time Minimization 

ED 


Table VI 

Objective Functions - Resource Utilization Maximization 
Group 


Objective Function 

References 

Elasticity Maximization 

ED 

Maximum Average Utilization Minimiza¬ 
tion 

(D, ED, ED 

Resource Utilization Maximization 

(H, (1), (H, ED, 
ED, H, (S) 

Resource Wastage Minimization 

ED, ED, (D 


A. Deterministic Algorithms 

Deterministic approaches are also proposed for the VMP 
problem, representing 19.0% of the studied articles. Particular 


methods of this solution technique are presented in Table VIII 


B. Heuristics 

In this work, 67.8% of the studied articles proposed 
heuristic-based solution techniques for the VMP problem. 
Particular methods of this solution technique are based on: 
(1) Best Fit Decreasing (BFD), (2) Based on Best Fit (BF), 
(3) Based on First Fit (FF), (4) Based on First Fit Decreasing 
(FFD), (5) Dominant Resource First (DRF), (6) First-Come, 
First-Served (FCFS), (7) Heaviest First (HF) and (8) Worst 
Fit-based (WF). Also (9) Greedy Algorithms and (10) Novel 
Heuristics are proposed (see Table |IX|i. 


C. Meta-Heuristics 


Performance Maximization, (5) QoS Maximization, (6) Re¬ 
source Interference Minimization, (7) Security Metrics Max¬ 
imization, (8) Shared Last Level Cache (SLLC) Contention 
Minimization and (9) Total Job Completion Time Minimiza¬ 
tion (see Table 0. 


From the studied articles, 15.5% proposed the solution of 
the VMP problem with meta-heuristics. Particular methods 
of this solution technique are: (1) Ant Colony Optimization 
(ACO), (2) Evolutionary Algorithm (EA), (3) Genetic Algo¬ 
rithm (GA), (4) Memetic Algorithm (MA), (5) Neighborhood 


E. Resource Utilization Maximization 

Cloud infrastructures are commonly composed by multiple 
physical and virtual resources such as CPU, RAM, storage, 
network bandwidth and Graphical Process Units (GPU). In 
this context, an efficient and balanced utilization of these 
resources is an important issue to address. Taking into account 
the surveyed universe, 15.5% of the studied articles proposed 
the optimization of resource utilization. Objective functions 
classified in this optimization group are: (1) Elasticity Maxi¬ 
mization, (2) Maximum Average Utilization Minimization, (3) 
Resource Utilization Maximization and (4) Resource Wastage 
Minimization (see Table |Vl|. 

VI. SOFUTiON Techniques 

In the considered universe, there exist different techniques 
proposed for solving the VMP problem. The main solution 
techniques include (1) deterministic algorithms, (2) heuristics, 
(3) meta-heuristics, and (4) approximation algorithms. The 
mentioned solution techniques are detailed in the following 
sub-sections and summarized in Table |21 


Table VII 

Objective Functions 


Objective Function 

Percentage 

References 




El, 

(D, 

(13, ED, 

ED, 



(78), 

(19), 

(28), 

ED, (D, 

(20), 

Energy 


ED 

ED, 

ED, 

El, El, 

(68), 

Consumption 

50.0% 

(38), 

(13), 

(71), 

El, ED, 

167), 

Minimization 


(80), 

(151, 

(87), 

(ID, ED, 

(431, 



(86), 

(37), 

ED, 

ED, ED, 

ED, 



ED, 

(3D, 

ED, ED, ED, (H 
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(^ 

(ID, 
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03, 

Network Traffic 
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(13), 
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ED, (3, 
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(D, ED, ED, ED, ED 
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(13, 

m. 
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ED, 
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ED, 
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ED 

ED, 

ED, 

ED, ED, 
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Performance 
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(SI, 

ED 

ED, 

(H, ED, 

El 

16.7% 

(33, 

(H 
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23, ED, d, (81] 

,(D, 

Resource 

Utilization 

Maximization 

15.5% 

(H, (9), (D, ED, ED, ED, (SD, 
ED, ED, ED, (D, El, ED 






































Table VIII 

Solution Techniques - Deterministic Algorithms 


Table X 

Solution Techniques - Meta-Heuristics 


Technique 

References 

Binary Integer Programming (BIP) 

ED 

Constraint Programming 

El, ED 

Convex Optimization Theory 

ED 

Dynamic Programming 

ED, ED 

Integer Linear Programming (ILP) 

El 

Linear Programming (LP) 

ED 

Matrix Transformation Algorithm 

ED 

Mixed Integer Linear Programming (MILP) 

ED 

Primal-Dual (Hungarian) Algorithm 

El 

Pseudo-Boolean Optimization (PBO) 

ED, ED 

Relationship Rules Framework 

ED 

Stochastic Integer Programming (SIP) 

ED 

Transverse Matrix Approach 

ED 


Table IX 

Solution Techniques - Heuristics 


Heuristic 

References 

Based on Best Fit Decreasing (BFD) 

ED, ED, ED, ED 

Based on Best Fit (BF) 

ED, ED 

Based on First Fit (FF) 

ED, ED, ED 

Based on First Fit Decreasing (FFD) 

0, ED 

Dominant Resource First (DRF) 

ED 

First-Come, First-Served (FCFS) 

ED 

Greedy Algorithms 

ED, ED, 0, ED, 
m, m, (201, ED, 
ED 

Heaviest First (HF) 

(23 

Worst Fit-based (WF) 

ED 

Novel Heuristics 

ED, ED, ED, ED, 
ED, ED, 0, ED, 
ED, 0, ED, ED, 
ED, m, (nl, ED, 
(2D, ED, ED, ED, 
ED, ED ra, ED, 
ED, 0 ED, ED, 
ED, ED, ED ED, 
El, El, ED, E3, 
El, ED, ED 


Search (NS), (6) Particle Swarm Optimization (PSO), (7) 
Simulated Annealing (SA), (8) Cut-and-Search (CS) and (9) 
Tabu Search (TS) (see Table 

D. Approximation Algorithms 

Heuristics and meta-heuristics provide good quality solu¬ 
tions, but quality of the expected solutions is hardly measu¬ 
rable. In a p-approximation algorithm, the value of a solution 
will not be more (or less) than a factor p times the optimum 
solution. Only 2.4% of the studied articles proposed approxi¬ 
mation algorithms for solving the VMP problem m, ill- 

VII. Formulation: Offline or Online 
The VMP can be formulated as online or offline problems. 
According to this review, VMP formulations mostly are pro¬ 


Meta-Heuristic 

References 

Ant Colony Optimization (ACO) 

ED 

Cut-and-Search (CS) 

ED 

Evolutionary Algorithm (EA) 

(6D 

Genetic Algorithm (GA) 

ED, 0, ED 

Memetic Algorithm (MA) 

El, eD, ED 

Neighborhood Seai'ch (NS) 

ED, El 

Particle Swai'm Optimization (PSO) 

ED, ED, 0 

Simulated Annealing (SA) 

ED, ED 

Tabu Search (TS) 

ED 


Table XI 

Solution Techniques 


Solution Technique 

Percentage 

References 

Deterministic 

Algorithms 

19.0% 

ED, ED, ED, 0,0, ED, ED, 
ED, ED, ED, ED, ED, 0, 
ED ED, ED 



m, 0, (H, (H, ED, ED, 



ED, ED, ED, ED, 

0, ED, 



ED, ED, ED, ED, 

4D, ED, 



ED, ED, ED, ED 

H, (21, 

IT 


ED, ED, ED, ED, 

801, l36l. 



ED, ED, ED, ED, 

53l, ED. 



ED 0, ED, ED, 

ID, ED, 



El, ED, 0, ED, 

4l. I50l. 



ED, ED, ED, ED, 

ID, ED, 



El, (H, El 




ED, 0, ED, (ID, ED ED 

Meta-Heuristics 

15.5% 

El, ED, (Hi, ED, 

SI, Ell, 



ED 


Approximation 

Algorithms 

2.4% 

E3, ED 


Table XII 

Eormulation: Oefline or Online 


Formulation Approach 


Online 


Offline 


Percentage 


77.4% 


22 . 6 % 


References 


Ha, ED, GD, ED. Eo|. 

ED, EH, El, (H, Ei, 

ED, (26), (H, (H, eD 

(29), EU, (D, ED, ED, 

El, El, ED, ED, m, 

El, ED, ED, El, ED, ED, 
ED, ED, ED, ED, E3 

ED, ED, ED, ED, ED, 

ED, El, ED, ED, ED 

ED, ED, ED, ED, ED, 

ED, ED, ED, ED, ED, 

ED, ED, ED, ED, ED 

El, El, m, E 


ED, El, ED, ED, ED 

ED ED, ED, 0, ED 

l28l . ED, ED, ED, ED, 

ED, ED, ED, ED 


posed as online problems (see Table |XII| i, considering the 
dynamic nature of cloud computing workload. 

For online formulations, workload prediction and forecas¬ 
ting techniques are proposed in 1921 . Il24ll . l80l . l67ll . 1^ . 

m, uni, ill, 0,139], isi, m, isa, eui, in, ed. 
























































Table XIII 

Cloud Architectures 


Table XIV 

Orientation: Provider or Broker 


Cloud Architecture 

Pereentage 

References 



O, (m, ED, 

El. El, 



f30l. f3l. I26l. (78 

, El El, 



(69), (21], (66), 

39l. (33. 



(40), (55), El, (32 

, ED, El, 



(21, m, ED, 

2D, (93, 



(77), El. (H. 

23. (H, 



(13, (23, (23, 

ID. E3, 

Single-Cloud 

88.1% 

(H, El, (13, 

43, (43. 



El, ED, E3, 

13. ED, 



(60), (23, EH. 

m, 0, 



Ea, El, (H, 

23, (23, 



(H, (13, ED, 

13. E3, 



m, m, ED, 

m ESI, 



El, ED, El, ED, El, 
(23, (13, ED, Ell 

Multi-Cloud 

10.7% 

E), (23, ED, ESI, El, 
ED, El 

Federated 

1.2% 

ED 


VIII. Cloud Architectures 

According to this review, the VMP problem have been 
studied in 3 different cloud architectures: (1) single-cloud (i.e. 
only one cloud computing datacenter), (2) multi-cloud (i.e. 
many cloud computing datacenters from one o more cloud 
service providers), and (3) federated clouds. The most studied 
cloud architecture is the single-cloud (see Table [Xln) i. 

IX. Orientation: Provider-oriented or 
Broker-oriented 

Cloud infrastructure optimization is a main concern for 
CSPs, considering the objective functions described in Section 
[V| In this context, the VMP problem is commonly formulated 
from a provider-oriented perspective 15^ . 

Considering that placement decisions only include tenants 
constraints commonly represented as SLAs, tenants cannot 
decide which PMs will hosts their VMs. However, the number 
of CSPs has been rapidly increased and nowadays there are 
different pricing schemes, virtual machine offers and features. 
In general, it is difficult for users to search cloud prices and 
decide where to host their resources. A scenario for the VMP 
problem for optimizing users virtual infrastructure placements 
across available public cloud offers can be also studied. This 
novel scenario can be described as a broker-oriented approach 

CD. 


X. Experimental Environment 

Observations about the experimental environment conside¬ 
red by each studied articles are presented in this section, (1) 
focusing on what type of experiment proposed (i.e. simulation 
or implementation in real applications), (2) workload type 
(i.e. real or synthetic), (3) workload distribution, and (4) 
experiment size (i.e. number of PMs and VMs). 

Simulations are widely used in the studied articles, and 
some works even proposed a hybrid environment conside¬ 
ring simulations and complementing those simulations with 


Orientation 

Pereentage 

References 

Provider-oriented 

97.6% 

ED, ED, (23, (H. E3, 

El, ED, El, (D, (23, 

(13, El, El, El. (21, 

Ea, El, El, E3. (23, 

El, E3, ED, El (3 

El, El, El, (1, El 

El, E3, (13, (21, (H, 

0, (H, (H, (13, 0, (H, 
(H, (21, (21, El. E3, 

(2D ED, E3 (H. ED, 

(23, (H, El, E3, m, 

(21, 0, (13, El, E3, 

Ea, E3, E3, E3. (23 

(H, (23, m, E3. (21, 

ED, El, E3, E3, E3, 

(H, (H, 0, (D, E3, (2D 

Broker-oriented 

2.4% 

ED, ED 


Table XV 

Articles considering simulations or implementation in real 

APPLICATIONS IN THE STUDIED UNIVERSE OF 84 PAPERS. 


Experiment Type 

References 


(33, 

ED, 

(23, El, 

E3, 

El, 

El, 


(63. 

El. 

23, El, 

El 

Eo), 

(35). 


El, 

f^. 

ED, (23, 

0, 

El. 

(83, 


(22), 

(90), 

(21, (23, 

(37), 

(86), 

(30), 


1^, 

(53, 

El, El, 

(ia. 

(39), 

(53), 

Simulation 

(81 

87|, 

23, E3, 

|12|, 

(38), 

(49), 

(lO). 

(741. 

ED, El, 

(191. 

(201. 

(291, 


ED, 

El, 

(33, (S3, 

E3 

(52) 

0, 


(56|, 

(47), 

(Sa, 0 , 

(83), 

(68), 

(23), 


(80), 

E3, 

El, El, 

(ll. 

l3l|. 

(15), 


(23, 

0, El, El, El, 0 

(21, 

(23, 


(Ml 






Applications 

El, ED (21, (13, El 

Not Identified 

E3, (2D 


implementations in cloud operating systems to validate the 
feasibility of the proposal in real scenarios ED, im, 1321, 
ll59l . Simulations can include experimental tests with different 
numbers of VMs and PMs, different types of workloads (e.g. 
CPU-intensive, network-intensive) and different distribution of 


workload (e.g. Gaussian or normal distribution). Table XV 


presents studied articles that proposed both simulations and 
implementations in real applications, while Tables |XVI| and 


XVII present workload types and distribution proposed in 


the studied articles. Einally, Eigure classifies articles by 
experiment size. 

Most of the studied articles assume homogeneous resource 
conhgurations for PMs [33]. CD, Q, [851, US), HD, ES, 

1, ED. 


XL Conclusions 

The presented work reviewed a large part of the VMP 
literature considering a well-dehned selection process of re¬ 
search articles. Articles are classihed considering: (1) opti¬ 
mization approaches, (2) objective functions and (3) solution 
techniques. 



























Table XVI 

Articles considering synthetic or real workload in the 
STUDIED universe OF 84 PAPERS. 


Workload Type 

References 

Synthetic 

dl], (66j, (60), (61], 0, (42), (m, 
(73], dD, (88], (7), (40), (85], (13, 
(90). 1^. 1^. (37], (86], (30), (55), 
flTl. (33], (39), (53], 0, (13, (28), 

(23, do], (H, (13, (H, (13, d3, 
(13, (3, m], (23, (3, (13, El, E3, 
(36], (59], (13, dl], (23, (3, (H, 
(79], (9], (76], (23, (13, do] 

Real 

(23, dl, (25], (H, (50], (26], (69], 
(H], (65), (13, (H, (75], (23, do] 

Not Identified 

(13, (13, (13, dl], (13, d3, dl], 
El], (13, (13, (23, (23, (H, do], 

(H 


Table XVII 

Articles considering difeerent workload distributions in the 
STUDIED universe OF 84 PAPERS. 


Workload Distribution 

References 

Random 

dS, (13, (13. d3. (13. (23. 123. 
(23. 0. m]. (23 

Uniform 

dD. da. (13. 0 . (23. (13. da. 
(23, 0 

Normal 

(13. (13. d3. da 

Poisson 

0 , (13, da 

Gaussian 

dl 

Custom 

m 

Not Identified 

d3. (23. (13. (13. (13. (23. (23. 
(13, dl, (23, (13, (23, (23, ( 13 , 
d 3 , 0 , (23. (13. (23 (23. (13. 

l44i. Eg, (13, (23, d3, ( 93 , (13, 

( 2 a. EK. d3. (23. (23. (23. (23. 
da. (23. (23. dl. (23. da. da. 
(23.0. (23. (23 (13.0 (13. (13. 
(23, dg, (23, (13, (23. (23. (23. 
(23. 0. da. (13 


100-1000 VMs 



Figure 4. Percentage of articles considering number of VMs used for 
experiments in the studied universe of 84 papers. 


Additional identified classification criteria could be: (1) if 
the VMP problem is studied online or offline, (2) which cloud 
architectures are considered, (3) if the VMP is broker-oriented 
or provider-oriented and (5) what type of experimental envi¬ 


ronment is proposed. 

Considering the classification and statistical data presented 
in this work, a deeper study of the selected articles should be 
performed as future direction of this PhD research. 
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